Skip to content

whisper : mask padded K/V in flash-attention encoder and cross-attention#3941

Open
gooofy wants to merge 1 commit into
ggml-org:masterfrom
gooofy:fix/flash-attn-pad-mask
Open

whisper : mask padded K/V in flash-attention encoder and cross-attention#3941
gooofy wants to merge 1 commit into
ggml-org:masterfrom
gooofy:fix/flash-attn-pad-mask

Conversation

@gooofy

@gooofy gooofy commented Jul 16, 2026

Copy link
Copy Markdown

With flash attention enabled, the encoder self-attention and the decoder cross-attention view their K/V buffers padded to GGML_PAD(n_ctx, 256) but only write n_ctx rows, and call ggml_flash_attn_ext with a null mask. The unwritten pad rows are attended at full weight; their content is whatever the buffer holds - zeros on a fresh state, or leftovers of previous whisper_full calls with a different audio_ctx (layer strides il*n_ctx_pad shift between window sizes, so runs overwrite each other's pad regions).

On a long-running server this makes transcriptions depend on request history: e.g. large-v3-turbo with audio_ctx=250 on a 3 s clip produced 5x sentence duplication on a fresh state, 2x after one audio_ctx=500 request, and clean output after full-window requests. The default full window is affected too (36 unmasked rows out of 1536), just diluted enough to go unnoticed.

Fix: give both call sites a proper mask input (mirroring the decoder self-attention KQ_mask) with -INFINITY over [n_ctx, n_ctx_pad), filled in whisper_encode_internal / whisper_decode_internal. Outputs at reduced audio_ctx are now deterministic regardless of prior requests; timing is unchanged.

With flash attention enabled, the encoder self-attention and the decoder
cross-attention view their K/V buffers padded to GGML_PAD(n_ctx, 256)
but only write n_ctx rows, and call ggml_flash_attn_ext with a null
mask. The unwritten pad rows are attended at full weight; their content
is whatever the buffer holds - zeros on a fresh state, or leftovers of
previous whisper_full calls with a different audio_ctx (layer strides
il*n_ctx_pad shift between window sizes, so runs overwrite each other's
pad regions).

On a long-running server this makes transcriptions depend on request
history: e.g. large-v3-turbo with audio_ctx=250 on a 3 s clip produced
5x sentence duplication on a fresh state, 2x after one audio_ctx=500
request, and clean output after full-window requests. The default full
window is affected too (36 unmasked rows out of 1536), just diluted
enough to go unnoticed.

Fix: give both call sites a proper mask input (mirroring the decoder
self-attention KQ_mask) with -INFINITY over [n_ctx, n_ctx_pad), filled
in whisper_encode_internal / whisper_decode_internal. Outputs at reduced
audio_ctx are now deterministic regardless of prior requests; timing is
unchanged.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant